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Abstract: This paper describes use of an online Student Response System (SRS) in a pre-qualification course for engineering 
studies in Norway. The SRS in use, where students answer quizzes using handheld mobile devices like Smartphones, PADs, 
iPods etc., has been developed at Spr-Trpndelag University College. The development of the SRS was co-funded by the 
Lifelong Learning Program KA3-ICT in 2009-2010. SRS has been designed to help teachers effortlessly i) break the 
monotony of a lecture and allow the students to actively take part in the lecture, ii) increase teacher-student interaction, 
and iii) give teacher and students immediate anonymous feedback on learning outcome. The response system was used in 
mathematics in two groups with different lecturers during two semesters in 2009-2010. The pedagogical methods in use 
will be referred to as "Peer Instruction" and "Classic". In each method the students will answer a multiple choice quiz using 
their mobile devices. In both cases the result of the quiz will immediately appear as a histogram on a screen in the 
classroom. The closing parts will also be identical. The lecturer then highlights the correct option in the histogram and 
explains why this option actually is the correct one. In the Peer Instruction method there will be an additional element. The 
first poll will be followed by a discussion in student groups, where the students are urged to defend their choice and 
convince their fellow students that their chosen option is the correct one. The discussion is then followed by a new 
individual voting session before the final results are shown and the closing part takes place. The paper will compare this 
method with the peer instruction method as described in existing literature. The learning outcome will be discussed 
according to interviews with students and the lecturers' experiences from the classroom. In addition we will analyze 
students' grades and test results in mathematics with respect to their expected level, based on previous achievements. We 
will present results showing that when students are arguing their point of view, they will have a stronger tendency to 
convince their fellow students when they themselves already have found the correct option in the quiz. Finally we will 
suggest pedagogical improvements for future use of response systems in mathematics. Input from lecturers and from 
students has already been used in the process of developing a new version of SRS, finished in January 2013. 

Keywords: student response systems, mobile learning, smartphones, peer instruction and learning, peer learning 
assessment systems, learning outcome 

1. Student response system (SRS) 

1.1 Technical equipment 

The SRS in use was developed at S0r-Tr0ndelag University College (HiST), cofounded by the 
European Commission (EduMecca 2009). In the current version the lecturer must be equipped with a 
computer, preferably connected to a projector. Each student needs a device capable of accessing a 
web page, for example a smartphone, iPod, iPad, laptop etc. In addition there must be a wireless 
network present. 

The major advantages with this system compared to traditional clickers are independence of 
software and flexibility in use of voting device. A quiz might be presented through any kind of 
computer application. The quiz might also be written on the fly, even on paper or a traditional 
blackboard. In the testing phase described in this paper we handed out iPods for a one year loan to 
all students, although some of the students preferred to use their own devices. In the following 
semester the students could choose between borrowing an iPod or using their own smartphone. As 
a result more than 80 % preferred to use their own equipment. The advantages of a system using 
equipment the students already own and are familiar with are obvious. The students' mobile phones 
are in this way transformed from an unwanted source of distraction into a powerful learning tool. 
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1.2 Voting procedure 

We will differentiate between two methods, Classic and Peer Instruction (PI). Figure 1 shows a 
flowchart for the two different variants of the SRS session. 

Flowchart for the SRS session 



Figure 1: Illustration of the SRS session 

In both methods the students were given a quiz, usually at the start at the lecture. They were given a 
couple of minutes to make up their opinion and optionally discuss the problem before casting 
individual votes. The result of the voting session would then appear as a histogram on the lecturer's 
computer and be projected onto the screen. The closing session is also similar in the two methods. 
The lecturer opens for a class discussion where the students are urged to defend their own choices. 
Finally the lecturer highlights the correct option in the histogram and explains why this is the correct 
one why the others are not. 

The lecturer can choose to arrange a second voting session involving a student discussion in groups 
consisting of three to five students. This option can be decided upon in advance or on the fly 
depending on the result of the first vote. 

The peer instruction method used here is a slight modification of the original method described by 
Mazur (Mazur 1997). Originally, it is seen as important that the students make individual decision 
before the discussions, where they are supposed to defend their own choice for solution and 
convince their fellow students of its validity. 
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Our reason for suggestion a different approach is mainly to add flexibility. It is usually not necessary 
to open for student discussion when most of the students agree on the correct solution. 

A similar method is suggested by Dufresne et al (Dufresne 1996), but without the lecturer's 
explanation before closure. 

1.3 Categories of quizzes 

We will differentiate between three categories of using SRS: 

■ Repetition: Testing the students' skills in a previous lecture's theme. 

■ Wake up: Testing the students' skills in the present lecture's theme. 

■ Introduction: Preparation for a new learning session. 

1.3.1 Repetition 

The majority of events have been in this category. A voting session in this category is usually held at 
the start of a new lecture. The basic idea is that if most of the students give the correct answer, it 
will not be necessary to give any further explanation, but rather move on to the next theme. If the 
result of the quiz shows that a significant part of the students does not choose the correct solution, 
this will be a strong incentive for the lecturer to return to the previous lecture and give a new 
explanation. This may be important particularly in subjects where the learning material is of a 
sequential nature, as in mathematics. This method will allow the lecturer to vary the lecture 
according to the students' response according to Draper's contingent teaching (Draper 2004). 

1.3.2 Wake-up 

The question of whether the students really are following the lecture should be the concern of any 
lecturer. The most common method is plainly to ask a question and wait for the students to raise 
their hands and answer the question. This will not usually give a good indication of the students' 
understanding, since only a few students will raise their hands, while the majority will remain 
inactive. Use of SRS provides a method to get answers from the entire group of students, since 
everyone is guaranteed anonymity. This feedback allows the lecturer to decide whether the lecture 
should continue as planned or if it is necessary, for instance give one more example or look at the 
theme from a different angle. 

1.3.3 Introduction 

In the third category student activity will be more important than the actual feedback. The desired 
feedback is that a new concept will be introduced to the students either through some sort of open 
question or a problem that can be solved combining existing knowledge in new ways. One example 
that we used was the question “The weight of one brick is one kilogram plus the weight of a half 
brick. What is the weight of one and a half brick?" as an introduction to equations. The solution can 
be found the hard way by trial and error, but is found with less effort for the ones capable of using 
mathematic language and set up the equation for the problem. It is interesting to note that only 12 
% managed to find the correct solution before they were introduced to the theory of equations, 
which should give a strong motivation to learn a systematic method for solving the problem. 
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2. SRS and mathematics 

From autumn 2011 the SRS has been in systematic use at the pre-qualification course at the Faculty 
of Technology at S0r-Tr0ndelag University College. The course is aimed at students from vocational 
schools who need this one year course to be allowed to study for a bachelor or master degree in 
technology. The annual acceptance of students has been 240 and the students are divided into four 
groups with approximately 60 students in each group. This study covers the subject mathematics in 
two groups during the testing period for 2010-11. The two lecturers involved were responsible for 
one group each, and had the responsibility for developing the quizzes in mathematics and physics 
respectively. This led to the establishment of a database consisting of 276 quizzes in mathematics 
and 89 quizzes in physics. 

All data presented are from one of the groups, while the lecturers' experiences and the result from 
student interviews are collected from both groups. The subject lasted for two semesters, and the 
students were given one lecture each day, consisting of 2x45 minutes. Usually the students were 
given one to three SRS quizzes in each lecture. 

Two thirds of the quizzes were held as Repetition, while the last third was almost equally divided 
into the Wake-up and the Introduction categories. Many of the quizzes were not consisting of 
conceptual problems, instead focusing on calculation techniques. 

2.1 Classic versus peer instruction 

There were a total number of 123 quizzes in the course, most of them using the Classic method as 
can be seen in figure 2. In most cases of Peer Instruction this method was decided upon after the 
result of the first voting. An unsatisfactory result gives the lecturer the opportunity to choose a 
student discussion and a new voting. Figure 3 shows the relative distribution of correct answers 
from each quiz. All quizzes that were expanded to PI have a result in the lower part of the 
distribution. When the method is decided upon according to the first voting, the students will see 
the result of the first voting, which is not recommended in earlier literature (Mazur 1997). It is 
important to notice that the SRS is compatible with traditional Peer Instruction. This method can be 
chosen in advance to ensure that the result of the first voting sessions will not be shown until the 
last voting is finished. Then both results will be sequentially displayed. 


Classic versus PI 

■ Classic b PI 

80 
60 
40 
20 
0 

1st semester 2nd semester 

Figure 2: Different types of quizzes 
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Percentage of correct answers 
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Figure 3: Correct answers in classic versus peer instruction 


2.2 Improvement after peer instruction 

Figure 4 shows the number of correct answers before and after the discussion for the 14 cases 
where Peer Instruction was used. The percentage of correct answers did improve in all quizzes 
except in one. The average percentage of correct answers in the first voting session was 34 %, while 
the number increased to 61 % in the second session. These results strongly support our hypothesis 
that the students with the correct solution are more likely to convince fellow students to agree with 
their point of view than the other way around. Earlier research on Peer Instruction in physics 
teaching for college students (Crouch 2001) also gives similar results. 

The student interviews suggest a preference for peer instruction. "You have the time to make up 
your own opinion before the discussion. Then you compare your own answer with the student 
sitting next by. You compare your answers and then you agree on a final answer together." "And it 
might happen that if you have a theory, it will be proven to be false. And then it perhaps is easier to 
remember when you do the tests." 


Percentage of correct answers i 1st 
and 2nd voting session 
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Figure 4: Increase in correct answers in 2nd voting session 


Even though PI is a very effective teaching method, we will still emphasize the importance of the 
lecturer's explanation before closing the session. The students identify the explanation as the most 
important part of the session in interviews (Hansen-Nygard 2011). A research among biology 
students actually shows that the combination between discussion in student groups and the 
lecturer's explanation will produce maximum learning outcome (Smith 2011). 


3. Experiences from use of SRS 

The students' opinions of SRS are collected from a poll involving all students and from group 
interviews. Three groups consisting of four students in each were interviewed. 
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We have registered a positive attitude towards SRS from the students. A significant majority of the 
students are of the opinion that SRS increases both engagement and learning. 


Figure 5 shows the distribution of students' evaluation of SRS' contribution to increasing 
engagement in classes. More than 70 % of the students claim increased engagement in classes 
where SRS was used. 


Students' view of SRS contribution to 
engagement 


70 % 



Very high High Neutral low 


Very low 


Figure 5: Results from student poll on engagement 


This is also confirmed in the student interviews. Student quotes: "It feels a bit like a small exam... 
Yes, I will pass it. So, I try to do my best.", "...And you suddenly wake up instead of just waiting for 
the lecture to end. So it definitely helps in raising my spirit". 

The lecturers also report increased student activity. Starting the lecture with a SRS session is found 
to provide engagement and participation throughout the entire lecture, not only the SRS session. 

Figure 6 shows student responses to the hypothesis "SRS increases my learning". Almost 90 % agree, 
while none of the students disagree. 
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The fact that students claim higher learning outcome in polls does of course not guarantee that this 
really takes place. There have been many studies on the effect of response system on learning 
outcome measured by tests and exams. 

Mazur reports significant improvements in learning outcome. This is supported by other studies 
(Caldwell 2007), while others show small improvements (Chen 2010). Some studies still show a 
neutral effect (James 2012). The variations in learning outcome suggest that the result is dependent 
on other factors than just use of response systems. Factors to be considered are lecturer's choice of 
methods and relevant quiz questions and ability to motivate the students' participation and 
engagement. We find support for this in the student interviews: 

“It is like in mathematics, when almost everyone gets the right answer, there will not be much of a 
discussion afterwards. On the other hand, if only half of the class or even less are wrong there will 
be engagement and peer discussion." "In my opinion variation is the key. The A students have found 
their way in the old system with blackboard, lecturer and students. It is working for them. Variation 
in the classroom is the key." "Superficial questions don't do much for us. We already have too many 
of them." 

This study has focused on different level of the students based on their acceptance points, and we 
have found support for this hypothesis. 

According to the lecturers there seem to be an improvement in the students' calculation techniques 
leading to fewer primitive errors in assessments, even if there are no statistical evidence so far to 
confirm this assumption. An analysis of the final student grades including a control group not using 
SRS (Arnesen 2012) seem to indicate that students with lower grades from vocational school have a 
lesser tendency to fail the course in the SRS groups than in the control group. Since those students 
generally suffer most from lack of calculation technique, this result might be interpreted as a 
support for the assumption on improved skill in performing simple arithmetic operations 
In this study we had two groups with different lecturers using SRS and a third control group that did 
not use SRS. The two SRS groups differed in use of methods. The group analyzed in this paper 
differed from the second by more frequent use of Peer Instruction (16 quizzes) and more variation 
between Repetition, Wake up and Introduction. The other group had a more limited number of Peer 
Instruction (5 quizzes) where almost all quizzes were given as Repetition. The first group will be 
labeled SRS-P, and the second SRS-C. We also note that there were more quizzes given in SRS-P (126) 
than in SRS-P (87). 

The grading for the subject follows the ICTS standard, A being the highest grade and F representing a 
failed course. The average grading is obtained by giving 5 points for each A, 4 for each B and so on. 

An inspection of the final grades for the three groups as shown in table 1 does not strongly support 
any progress in student performance in the SRS groups. 


Table 1. Final grades in mathematics 


Group 

A 

B 

C 

D 

E 

F 

Total 

Percentage 

failed 

Average 

score 

Control 

7 

7 

10 

12 

1 

7 

44 

15,9 

2,7 

SRS-P 

2 

6 

11 

12 

1 

7 

44 

15,9 

2,2 

SRS-C 

7 

8 

8 

11 

7 

5 

46 

10,9 

2,6 


We observe that the control group actually has the best average grade score, while the number of 
failed students is lowest in one of the two SRS groups. 
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There are of course a number of factors affecting the grades, examples being performance of the 
lecturer, student effort and educational level in advance to the course. The most important and 
available factor is the earlier grading score from vocational school. This is converted to a number 
between 0 and 60, the acceptance points, when the students apply to Spr-Trpndelag University 
College. Figure 7 shows the distribution of the performance of the students in the pre-qualification 
course of 2008-2009 depending on acceptance points. As expected higher acceptance points seem 
to increase expected grades. The Pearson correlation between acceptance points and grade is 0.33. 

Grades by acceptance points 

5 

4.5 
4 

3.5 
3 

2.5 
2 

1.5 
1 

0,5 
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Figure 7: Grading depending on acceptance points 

The importance of acceptance points is even more significant for students' tendency to pass the 
course. Of the 268 students in 2008-2009 180 passed the course, 88 did not finish the course and 60 
did not pass the final exam. Figure 8 shows the distribution of students who did pass the course 
sorted by acceptance points. 



(35,4£>> [40,45> [45,M> I50,S5> (55,60> [60,65> [65,70> 



Figure 8: Passing ratio based on acceptance points 


We observe that the passing ratio increases considerably when the students' acceptance points 
exceed 50. When we compare the three groups we should also take the acceptance points into 
consideration. As can be seen from table 2 the students in the control group have a higher average 
level than the two SRS groups. 


Table 2: Average acceptance points 


Student group 

Average acceptance points 

Control 

42.3 

SRS-P 

40.2 

SRS-C 

40.0 
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The acceptance points become even more important when we consider the distribution shown in 



The control group is the only group without a fair share of students in the critical area below 35 
points where academic achievements tend to drop drastically (Arnesen 2012). We should then 
expect better results in this group, both in better grades and higher passing ratio. 

The student polls indicate that the students are of the opinion that SRS increases their learning 
outcome. In the student interviews there are some suggestions that students whose performances 
usually lies in the lower half, will benefit more from SRS than the better students. 

It is therefore of interest to examine the results for the students with the lowest acceptance points 
in the three groups. We restrict the sample to students with acceptance points below 40.The reason 
for this is that due to a minor change in the calculation procedure from 2008, points from 2011will 
be from 5 to 10 points higher than earlier. The students from 2011 with points below 40 would then 
be expected to be similar to the students from 2008 with points below 50. Those are the students 
we want to examine with increased learning outcome as SRS effect in mind. The results can be seen 
in table 3. 

Table 3: Results for students with acceptance points below 40 


Group 

Average acceptance 
points 

Average grades 

Relative failure 

Control 

37.5 

2.2 

0.21 

SRS-P 

35.1 

2.2 

0.11 

SRS-C 

34.5 

1.7 

0.33 


As can be seen from the average acceptance points and figure 9, the control group still has the 
"best" students, but the SRS group with the most frequent and most varying use of SRS has better 
performance. The last SRS group comes out with the lowest performance, but does also contain the 
students with the lowest acceptance points and therefore arguably the lowest potential for success. 
As a final inspection of which kind of students that benefits from use of SRS, we will look into three 
tests given throughout the semester. The students were graded on a percentage scale, which can 
give more information than the crude scale of ECTS grades. Table 4 shows the Pearson correlation 
between and acceptance points and percentage score on each test. 
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Table 4: Correlation between acceptance points and results 


Group 

Test 1 

Test 2 

Test 2 

Control 

0.56 

0.55 

0.35 

SRS-P 

0.05 

0.15 

0.11 

SRS-C 

0.31 

0.31 

0.29 


These results strongly support the theory that SRS helped improving results among the students 
with the lower acceptance points. All groups have a positive correlation, but it is considerably lower 
in the SRS groups, especially in the SRS-P with the most frequent use of SRS. 

The final class discussion before the lecturer's explanation is a version of the original peer instruction 
method (Mazur 1997). We have found that while few students usually will contribute to a discussion 
involving the entire class, these inhibitions seem to lessen when they are urged to defend their 
answer in a SRS session. We also have the impression that starting the lecture with the use of SRS 
will more easily get the students into a learning mode and to participate more actively in the entire 
lecture. 

Another important reason for having the class discussion is the opportunity to reveal the possibility 
that the correct solution is reached based upon erroneous reasoning or conceptual 
misunderstandings. Recent research (Nielsen 2012) shows that in some cases the majority of 
students will reach the correct answer based on a misconception of fundamental theories. The class 
discussion can then be an opportunity for the lecturer to address and correct these misconceptions. 
One specific challenge is when the presented problem consists of performing basic mathematical 
operations, as for instance solving an equation or an integral. In these cases it will often be easier to 
find the correct answer by testing all the options in the quiz by "backward" calculations rather than 
solving the problem independent of presented options. One way to avoid this is to hold back the 
options until the students are casting their votes. In this case the students will have to rely upon 
their calculated answer instead of testing the different options. When this procedure is used, many 
students will reach a solution not listed among the given options. Therefore, we suggest adding 
"None of the above" as the last option in these cases. Some of these quizzes then need to have this 
option as the correct answer for the students to be able to trust their answer even if it is not listed 
as a possible solution. 

Unfortunately we do register a decrease in the number of participating students during the two 
semesters. Figure 10 shows the development of participants from the first to the last quiz. 



Figure 10: Development in number of voting students 
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This can partly be explained by a decreasing number of students. While 61 students started, only 43 
finished the subject. In addition the students' tendency to skip lectures will increase throughout the 
semester. In some of the quizzes the number of present students was recorded, and figure 11 shows 
the percentage of participating students in these cases. 


Percentage of participants 
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70% 
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Figure 11: Development in percentage of voting students 


The 95 % confidence interval for the slope of the regression line is given by (-1.15 ± 1.77) %. The 
probability for a negative valued slope is 80 %, meaning that the hypothesis of a possible decline in 
the percentage of participation is not statistically verified. The statistics for the regression line are 
shown in table 5. 


Table 5: Regression for development of participation 



Coefficients 

Standard 

Error 

t Stat 

P-value 

Lower 95% 

Upper 95% 

Intercept 

0,71544556 

0,06564733 

10,8983186 

3,9501E-12 

0,58155694 

0,84933418 

Slope 

-0,00114511 

0,0008703 

-1,31576487 

0,19790098 

-0,0029201 

0,00062988 


Another interesting angle is to compare the amount of participation during the two semesters. The 
data show a decrease from 66 % to 62 %. There is no statistical evidence for a difference between 
the two samples, This is seen in the statistics in table 6, 


Table 6: t-test on difference in participation by semesters 

1st semester 2nd semester 


Mean 

Variance 

Observations 

Hypothesized 

Difference 

df 

t Stat 

P(T<=t) one-tail 
t Critical one-tail 


0,65726382 0,61642325 

0,02547268 0,02146644 

16 17 

Mean 0 

30 

0,76437401 

0,2253076 

1,69726089 
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Still there is a fact that a significant part of the group is not participating. Explanations may be 
technical problems or lack of interest from the students. One of the interviewed students suggested 
a combination of these factors: "Some time you can blame it on technical problems. You might get 
an error message or you don't have any battery power left or you just have left the iPod at home. But 
sometimes you are just not up to it. You just don't feel motivated". Another possible explanation 
given in the interviews is that they often will choose not to answer if they do not have any preferred 
options: "/ think they don't answer because there are no "Don't know" option. People just don't want 
to make a wild guess, because if they are lucky and get the correct answer, the lecturer will get the 
impression that everyone has learned the stuff and just carry on" 

This last factor can be examined statistically. When the option “Don't know" is present, the mean 
participation increases from 60 % to 68 %. The hypothesis that inclusion of "Don't know" increases 
participation is still not statistically valid within a significance level of 0.05. As can be seen from table 
7, the actual significance level is 0.08 


Table 7: Relevance of "Don't know" option 

With «Don't know» Whithout «Don't know» 


Mean 

Variance 

Observations 

Hypothesized 

Difference 

df 

t Stat 

P(T<=t) one-tail 
t Critical one-tail 


0,678662254 

0,0223179 

14 

Mean 0 
28 

1,432469891 

0,081540007 

1,312526782 


0,603140112 

0,022523249 

19 


4. Emerging technologies 

A new version of the SRS is finished in January 2012. The most notable changes are a new and 
improved user interface, allowance for more multiple users and access to earlier voting results. 

The Done-IT project, cofounded by the European Commission, aims at developing an assessment 
system based on the described response technology using smartphones or pods (Done-IT 2011). The 
pedagogical idea is to change the assessment into an area for learning as well as evaluation. This can 
be achieved by using SRS technique for immediate feedback on student scores to individual 
questions. If one or more of the questions are unsatisfactory, the lecturer may then give a short 
lecture, allow group discussion etc. followed by a new voting session giving the students a second 
chance to improve their assessment (Thorseth 2012). The assessment system will be tested in the 
pre-qualification course during the autumn semester 2012. 


5. Conclusion 

Studies have shown that SRS either has benign or neutral effect on learning outcome. We have 
suggested three methods for use of SRS in mathematics, Repetition, Introduction, and Wake up. All 
methods depend on the method introduced by Mazur. We have focused on what type of students 
will benefit from SRS, and our findings support the theory that SRS will increase learning outcome 
among students who usually have academic achievements in the lower part of the group. We still 
emphasize that this may not be true in other subjects and for other student groups. The varying 
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results from different studies indicate that more research is needed to find and systemize the factors 
of success for use of clickers in education. 
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